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Implications  for  Linking  Job  Performance  and  Aptitude 

Paul  W.  Mayberry 
Center  for  Naval  Analyses 

^  The  selection  of  test  content  and  the  construction  of  job  performance 
tests  have  not  followed  any  specific  methodology.  Test  content  has  often 
been  limited  to  tasks  that  are:  most  frequently  performed  on  the  job,  easily 
tested  in  the  hands-on  mode,  or  selected  to  maximize  differences  in  individ¬ 
uals’  performance.  The  problem  with  such  test  development  procedures  is 
that  the  resulting  score  scale  allows  for  little,  if  any,  independent  interpreta¬ 
tion  of  the  test  scores,  other  than  normative  comparisons.  For  the  purpose 
of  validating  the  Armed  Services  Vocational  Aptitude  Battery  (ASVAB), 
such  relative  interpretations  may  suffice.  But  for  determining  qualification 
standards  on  the  ASVAB,  a  more  absolute  interpretation  is  required  of  the 
performance  test  scores.  Therefore,  a  systematic  methodology  is  necessary 
for  job  performance  test  construction  so  that  meaningful  score  interpreta¬ 
tion  and  linkage  with  ASVAB  will  be  possible. 

Significant  research  has  been  conducted  in  the  achievement  testing  arena 
that  provides  some  guidance  for  the  development  of  job  performance  tests, 
particularly  work  on  domain-referenced  testing.  A  domain-referenced  test 
consists  of  a  sample  of  tasks  from  a  well-defined  population  of  tasks,  such 
that  the  sample  estimates  the  proportion  of  tasks  that  an  individual  would 
be  expected  to  achieve  in  the  population.  This  definition  contains  two  es¬ 
sential  components:  specification  of  the  task  domain  and  appropriate  sam¬ 
pling  of  tasks  from  that  domain  so  that  the  part  will  represent  the  whole. 
Given  that  these  two  conditions  have  been  met,  a  domain- referenced  test 
measures  an  individual’s  degree  of  knowledge  on  a  continuum  of  no  pro¬ 
ficiency  to  complete  proficiency.  For  our  application  to  the  measurement 
of  job  performance,  we  define  this  continuous  scale  as  a  competency  scale. 
More  specifically,  competency  is  defined  as  the  percentage  of  tasks  correctly 
performed  in  the  full  task  domain  which  is  estimated  by  the  percentage  of 


tasks  correctly  performed  on  the  test. 

Our  definition  of  competency  does  not  focus  on  the  division  of  the  con¬ 
tinuous  scale  into  discrete  qualitative  categories  (e.g.,  competent  versus 
incompetent,  or  master,  journeyman,  apprentice,  etc.).  Rather,  our  focus 
is  on  developing  an  objective  and  reasonable  procedure  with  explicit  rules 
for  scale  construction  that  will  allow  for  direct  interpretation  of  the  result¬ 
ing  test  scores.  It  should  be  noted  that  the  domain-referenced  approach  to 
developing  a  competency  scale  does  not  preclude  the  setting  of  standards. 
Therefore,  to  the  extent  that  the  conditions  of  domain  specification  and 
task  sampling  are  upheld,  the  resulting  score  scale  of  the  test  can  be  inter¬ 
preted  in  an  absolute  mannerTAVe  now  turn  to  examining  these  two  critical 
components  for  constructing  a  competency  scale  for  the  hands-on  tests  for 
the  Marine  Corps  infantry  occupational  field. 

SPECIFICATION  OF  THE  TASK  DOMAIN 

For  the  purpose  of  relating  job  performance  test  scores  to  ASVAB,  the 
performance  test  must  first  be  firmly  based  on  a  detailed  specification  of 
the  job  requirements.  Within  the  military  community,  extensive  research 
has  been  conducted  on  the  definition  of  jobs  and  the  delineation  of  their 
specific  requirements  and  responsibilities.  This  research  provides  an  excel¬ 
lent  foundation  for  explicitly  defining  the  tasks  that  are  required  in  any 
job. 

Individual  Training  Standards 

Within  the  Marine  Corps,  the  Individual  Training  Standards  (ITSs)  de¬ 
fine  the  tasks  that  are  required  of  each  Marine,  the  level  of  competence 
to  which  the  tasks  must  be  performed,  and  what  stage  in  the  career  of 
the  Marine  this  competency  is  expected.  The  ITSs  serve  as  the  definitive 
statement  of  an  MOS’s  job  requirements  because  they  are: 

•  Comprehensive:  a  thorough  process  synthesizing  multiple  sources  of 


information  (e.g.,  occupational  surveys,  task  analyses,  job  experts, 
subject  matter  experts,  training  materials,  etc.)  which  receives  sig¬ 
nificant  review  and  verification  from  the  field. 

•  Independent:  job  requirements  are  defined  independent  of  the  JPM 
Project  and  therefore  are  not  necessarily  biased  towards  including 
only  those  requirements  that  are  easily  tested  or  would  maximally 
discriminate  between  individuals. 

•  Reproducible:  a  published  Marine  Corps  Headquarters  standing  oper¬ 
ating  procedures  (SOP)  exists  to  document  the  entire  developmental 
process  of  the  ITSs  and  to  allow  for  replication. 

•  Official  Marine  Corps  policy:  the  ITSs  are  completely  staffed,  re¬ 
viewed,  and  approved  by  Marine  Corps  Headquarters,  the  Division 
Commanders,  the  training  schools  and  institutes,  and  the  Fleet  Marine 
Forces,  so  that  the  ITSs  become  the  doctrinal  statement  of  the  job 
requirements  for  an  MOS. 

The  ITSs  are  central  to  the  Marine  Corps  JPM  Project.  If  ITSs  are  not 
available  for  a  particular  MOS,  the  Marine  Corps  would  not  consider  that 
specialty  for  the  development  of  job  performance  tests. 

The  ITS  tasks  for  the  infantry  occupational  field  are  organized  by  MOS 
and  sorted  into  relatively  homogeneous  duty  areas  within  each  MOS.  The 
duty  areas  and  associated  number  of  tasks  for  the  five  infantry  MOSs  to 
be  tested  in  the  Marine  Corps  JPM  Project  are  given  in  appendix  A.  For 
a  given  task,  the  ITSs  are  composed  of  the  following  components: 


Task.  Description  of  what  the  Marine  needs  to  be  able  to  do  in  com¬ 
bat.  The  task  has  a  specific  beginning  and  ending,  and  is  a 
logical  and  necessary  unit  of  performance. 

Administrative  instructions.  Directions  for  test  administration  and 
the  degree  of  simulation  necessary  to  measure  the  task. 


Training  objective. 

Behavior.  The  action  to  be  performed  by  the  Marine.  Some 
behaviors  are  identical  to  tasks,  which  implies  that  the  be¬ 
havior  can  be  performed  in  training  environments  just  as 
the  task  would  be  performed  in  combat. 

Conditions.  Equipment,  manuals,  assistance,  supervision,  spe¬ 
cial  physical  demands,  environmental  conditions,  and  loca¬ 
tion  that  pertain  to  performing  the  task. 

Standard.  Accuracy,  time  limits,  sequencing,  quality,  product, 
process,  etc.,  that  indicate  how  well  a  task  should  be  per¬ 
formed. 

Training  steps.  The  steps  that  must  be  performed  in  order  to  ac¬ 
complish  the  training  objective. 

References.  Manuals,  job  aids,  field  manuals,  etc.,  that  are  not 
listed  in  the  Conditions  section  that  will  guide  trainers,  instruc¬ 
tors,  or  performers. 

Table  1  presents  an  example  of  an  ITS  for  one  of  the  four  tasks  in  the  M203 
grenade  launcher  duty  area. 

In  addition  to  the  57  tasks  defined  for  the  thirteen  duty  areas  of  basic 
infantry,  the  ITSs  also  reference  Essential  Subject  Tasks  (ESTs)  that  are 
integral  to  performing  the  duties  of  am  infantryman.  The  ESTs  are  basic 
tasks  that  are  required  of  all  Marines.  Marines  are  tested  annually  on  a 
sample  of  these  tasks,  but  this  annual  qualification  is  used  as  a  training 
exercise  and  does  not  adhere  to  acceptable  measurement  principles.  The 
ESTs  that  extend  and  supplement  the  ITSs  are: 

•  First  aid  and  field  sanitation 

•  Nuclear,  biological,  and  chemical  defense 

•  Marksmanship 

•  Individual  tactical  measures 


TABLE  1 


INDIVIDUAL  TRAINING  STANDARD  FOR  ONE  TASK 
OF  THE  M203  GRENADE  LAUNCHER 


Task:  Engage  target  with  grenade  launcher. 

Administrative  instructions:  This  ITS  is  identical  to  the  actual  task. 
Targets  may  be  constructed  using  locally  available  material.  All  stu¬ 
dents  at  Infantry  Training  School  will  perform  this  ITS.  The  ITS  must 
be  performed  quarterly  by  grenadiers  in  conjunction  with  zeroing  the 
weapon. 

Training  objective: 

Behavior:  Engage  target  with  grenade  launcher. 

Conditions:  The  Marine  being  evaluated  is  provided:  M203  grenade 
launcher  and  seven  rounds  of  40mm  HE  DP  or  practice  ammu¬ 
nition  on  a  live-fire  range. 

Standard:  Obtain  hits  on  three  of  the  following  four  targets: 

•  From  a  kneeling  position,  place  a  round  through  a  window 
(.75  meters  wide  by  1  meter  high)  at  a  range  of  90  -  100 
meters. 

•  From  a  fighting  position,  hit  the  front  of  a  bunker  at  a  range 
of  135-150  meters. 

•  From  a  prone  position,  hit  within  5  meters  of  targets  in  an 
open  emplacement  at  a  range  of  275-300  meters. 

•  From  a  prone  position,  hit  within  5  meters  of  targets  in  the 
open  at  a  range  of  325-350  meters. 


TABLE  1  (Continued) 


Training  steps: 

•  Set  rear  sight. 

•  Establish  position. 

•  Aim. 

•  Fire  round,  sense  the  impact  of  the  grenade,  and  make  sight 
adjustment. 

•  Take  immediate  action,  if  necessary. 

References: 

•  FM  21-2,  Soldier’s  Manual  of  Common  Tasks,  Skill  Level  1,  pages 
3-114  through  3-119. 

•  TM  9-1010-221-10,  Operator’s  Manual,  40mm  Grenade  Launcher 
M203,  pages  18-22. 

•  FM  23-31,  M203  40mm  Grenade  Launcher,  chapter  6. 


Source:  Individual  Training  Standards  (ITS)  System  for  Infantry,  Occupa¬ 
tional  Field  (OccFld)  03,  MCO  1510. 35A. 


•  Land  navigation. 


Therefore,  the  ITSs,  supplemented  by  the  ESTs,  serve  as  the  official 
Marine  Corps  statement  defining  the  domain  of  job  requirements  for  the 
infantry  occupational  field.  Certainly,  the  ITSs  do  not  purport  to  specify 
all  the  possible  responsibilities  of  an  infantryman,  but  rather  document  at 
least  those  aspects  of  the  job  that  the  Marine  Corps  believes  to  be  most 
critical  to  the  functioning  of  an  infantryman  in  combat. 

Determining  the  Behavioral  Elements  of  Tasks 

By  themselves,  the  ITSs  provide  little  information  about  the  underly¬ 
ing  skills,  knowledge,  and  behaviors  required  of  the  tasks  that  compose  an 
MOS.  Sampling  of  test  content  with  regard  only  to  tasks  may  be  somewhat 
misleading  because  one  can  not  explicitly  identify  the  specific  behaviors 
that  are  being  measured  nor  assure  that  all  behaviors  relevant  to  the  per¬ 
formance  of  the  job  are  even  being  tested.  The  ultimate  inference  of  test 
scores  is  to  the  range  of  job  behaviors  required  for  a  duty  area,  not  to  the 
set  of  tasks  that  compose  that  duty  area.  Ultimately,  the  level  of  gener¬ 
alization  of  behaviors  will  be  to  the  MOS,  and  ideally  even  across  MOSs. 
Incorporating  behavioral  elements  into  the  test  construction  process  will 
help  to  facilitate  these  generalizations. 

To  identify  the  behaviors  inherent  in  the  tasks  of  the  ITSs,  the  Marine 
Corps  JPM  Project  conducted  task  analyses  and  relied  upon  job  experts  to 
construct  behavioral  elements  for  each  ITS  task.  The  behavioral  elements 
are  generic  verb-noun  statements  denoting  identifiable  units  of  performance 
that  underlie  the  performance  of  the  ITS  tasks.  In  this  regard,  the  behav¬ 
ioral  elements  provided  a  qualitative  basis  for  understanding  the  inherent 
behaviors  of  a  task  as  well  as  the  means  to  make  comparative  statements 
of  behavioral  similarity  across  tasks. 

Behavioral  element  by  task  matrices  were  constructed  for  each  duty  area 
of  the  infantry  occupational  field.  Figure  1  presents  the  matrix  developed 
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for  the  M203  grenade  launcher  duty  area.  The  five  tasks  that  compose  this 
duty  area  are  across  the  top  of  the  matrix  with  the  behavioral  elements  that 
underlie  those  tasks  in  the  leftmost  column.  The  diagonal  words  under  the 
tasks  correspond  to  the  training  steps  of  the  tasks,  but  are  not  necessarily 
the  training  steps  as  listed  in  the  ITSs.  Job  experts  reviewed  each  task 
to  produce  comparable  performance  units  that  were  on  the  same  level  of 
specificity  -  the  ITSs  training  steps  were  reworked  to  establish  these  units. 
Much  attention  was  devoted  to  this  equating  of  training  steps  because  they 
will  eventually  be  used  as  the  sampling  units  for  test  content. 

Marine  Corps  job  experts  at  two  of  its  three  Divisions  reviewed  the  ma¬ 
trices.  The  experts  examined  the  matrices  a  column  at  a  time  to  determine 
that  indeed  each  behavioral  element  was  required  of  that  training  step.  Be¬ 
havioral  elements  were  reworded  and  refined,  and  missing  behaviors  were 
added  to  the  list.  Next,  the  experts  reviewed  the  matrices  a  row  at  a  time 
to  ensure  that  the  behavioral  elements  were  comparable  across  the  training 
steps.  Elements  were  collapsed  or  expanded  as  thought  necessary  by  the 
expert  panel.  Finally,  the  matrices  were  reviewed  in  their  entirety  to  be 
certain  that  the  total  picture  completely  represented  the  duty  area.  Given 
the  approval  of  these  two  independent  panels  of  experts,  the  behavioral 
element  by  task  matrices  established  the  domain  of  job  requirements  for 
each  duty  area  in  terms  of  specific  tasks  and  common  behaviors. 

SAMPLING  TEST  CONTENT 

Resources  for  hands-on  performance  testing  are  limited  -  limited  in 
terms  of  personnel,  testing  time,  budget,  etc.  Given  that  the  entire  domain 
of  job  requirements  can  not  be  tested  due  to  these  constraints,  a  sample 
of  the  domain  must  be  selected  so  that  inferences  about  an  individual’s 
level  of  competency  can  be  made  from  his  performance  on  a  partial  set 
of  the  tasks.  Therefore,  in  addition  to  a  detailed  specification  of  the  task 
domain  and  the  associated  behaviors  that  define  a  job.  an  explicit  plan  for 
determining  which  specific  tasks  will  be  tested  is  necessary.  This  sampling 
strategy  involves  three  stages:  defining  the  sampling  unit,  establishing  the 
sampling  rules,  and  sampling  test  content  and  constructing  testable  units. 


Defining  the  Sampling  Unit 

Implicit  in  the  sampling  process  is  that  a  selection  is  being  made  among 
discrete  units  that  have  clearly  defined  boundaries  and  limits.  Also,  the 
units  are  considered  to  be  equal  or  interchangeable,  although  they  may 
have  different  probabilities  of  being  sampled.  These  conditions  pose  strict 
requirements  and  fundamental  difficulties  for  the  application  of  sampling 
techniques  to  job  requirements. 

Jobs  can  not  readily  be  decomposed  into  discrete  and  independent  units 
of  performance.  While  tasks  are  often  grouped  to  define  a  job,  the  collec¬ 
tion  of  tasks  is  often  arbitrary  or  a  matter  of  convenience.  Tasks  are  not 
natural  entities  unto  themselves,  but  rather  are  amorphous  groupings  of 
many  behaviors.  Selection  among  such  arbitrary  units  of  performance  can 
not  result  in  a  test  that  is  representative  of  the  whole  domain  of  job  re¬ 
quirements.  The  solution  involves  extensive  analyses  to  refine  the  tasks 
into  equivalent  performance  units  that  are  amenable  to  sampling. 

Establishing  the  equality  of  performance  units  -  or  defining  the  sam¬ 
pling  unit  -  is  based  completely  upon  the  input  of  job  experts.  This  use  of 
job  experts  for  definitional  purposes  should  not  be  confused  with  the  use  of 
experts  for  judgment-based  selection  of  tasks.  In  our  application,  the  job 
experts  are  used  to  delimit  the  boundaries  of  the  sampling  units  and  to  en¬ 
sure  their  equality,  not  to  determine  which  units  will  be  tested.  Therefore,  it 
is  our  contention  that  expert  input  is  required  only  on  matters  that  demand 
job  knowledge  or  could  benefit  from  their  past  experiences.  This  excludes 
selection  decisions  that  may  be  influenced  by  individual  preferences  or  bias. 
Selection  decisions  among  alternatives  judged  to  be  equivalent  by  job  ex¬ 
perts  is  best  accomplished  by  sampling. 

Given  the  scope  and  noted  arbitrariness  of  tasks,  the  Marine  Corps 
JPM  Project  focused  on  the  training  step  within  a  task  as  the  sampling 
unit.  Our  definition  of  the  training  step  as  the  sampling  unit  is  not  com- 


pletely  consistent  with  the  ITS  definition  of  a  training  step.  Based  upon 
extensive  expert  review  of  the  ITSs,  some  ITS  training  steps  were  collapsed, 
others  were  refined,  so  that  the  resulting  performance  units  were  considered 
comparable.  The  redefined  training  steps  were  thought  to  be  meaningful 
units  of  performance  that  maintained  their  fidelity  to  the  overall  task  while 
not  trivializing  the  performance  requirements.  Figure  1  shows  the  behav¬ 
ioral  element  by  task  matrix  that  defines  the  job  requirements  for  the  M203 
grenade  launcher.  The  training  steps  listed  under  each  task  were  judged  by 
job  experts  to  be  essentially  equivalent.  These  training  steps  will  serve  as 
the  units  to  be  sampled  and  thereby  determine  the  hands-on  test  content. 

Establishing  the  Sampling  Rules 

Having  obtained  equivalent  sampling  units,  systematic  and  defensible 
sampling  rules  must  be  made  explicit.  These  rules  explain  any  constraints 
within  which  the  sampling  must  be  conducted.  The  sampling  rules  for  the 
basic  infantryman  content  are  listed  below: 


•  All  duty  areas  defined  in  the  ITSs  for  the  basic  infantryman  MOS 
(0300)  in  pay  grades  E1-E3  will  be  tested. 

•  Total  testing  time  will  be  limited  to  6  hours.  Actual  testing  time  will 
be  up  to  250  minutes,  plus  up  to  110  minutes  of  administrative  time. 

-  The  actual  testing  time  for  any  duty  area  will  range  from  10  to 
30  minutes. 

•  Any  training  step  can  be  eliminated  from  the  sampling  process  for 
the  following  reasons: 

-  Hazardous  or  expensive:  includes  live  firing  of  most  weapons 
(except  the  M16A2  rifle)  and  crossing  a  contaminated  area  for 
the  NBC  duty  area. 

-  Expert  judgment  that  the  step  provides  little  or  no  information: 
applies  to  trivial  or  repetitive  training  steps  (this  does  not  trans- 
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late  into  the  exclusion  of  simple  training  steps). 


Requiring  all  basic  infantryman  duty  areas  to  be  tested  reflects  the 
Marine  Corps  position  that  all  duty  areas  included  in  the  ITSs  are  impor¬ 
tant  and  critical  to  the  performance  of  an  infantryman. 

Testing  time  is  certainly  the  most  dominant  constraint  on  the  sampling 
plan.  If  unlimited  testing  time  were  available,  the  full  domain  of  job  re¬ 
quirements  could  be  tested  in  each  duty  area.  To  the  extent  that  the  time 
required  to  test  the  full  domain  does  not  match  the  available  testing  time, 
the  reliance  on  sampling  becomes  more  important.  While  the  first  sam¬ 
pling  rule  states  that  all  duty  areas  are  important  (and  therefore  warrant 
testing  time),  it  does  not  address  the  issue  of  relative  importance  among 
duty  areas  (i.e.,  different  allotment  of  testing  time  across  the  duty  areas). 

Job  expert  ratings  of  importance  and  criticality  were  collected  for  the 
duty  areas  of  the  basic  infantryman  MOS  to  provide  a  basis  for  the  alloca¬ 
tion  of  testing  time.  The  ranking  of  duty  areas  according  to  these  ratings  is 
reported  in  table  2.  In  addition,  estimates  of  the  total  time  to  test  the  full 
domain  of  job  requirements  for  each  duty  area  are  noted  in  the  table.  In 
general,  the  importance  ranking  was  confirmed  by  the  number  of  training 
steps  and  behavioral  elements  associated  with  each  duty  area  -  that  is,  the 
more  important  duty  areas  tended  to  have  more  training  steps  and  behav¬ 
ioral  elements.  The  NBC  defense  and  first  aid  duty  areas  are  somewhat 
misleading  because  they  were  extensively  supplemented  by  the  ESTs. 

Based  upon  the  importance  ranking  of  table  2,  it  follows  that  testing 
time  should  be  allocated  to  duty  areas  accordingly  or  to  cover  the  full 
population  of  training  steps,  whichever  is  less.  The  full  domain  of  job 
requirements  can  be  tested  for  three  duty  areas:  SAW,  LAW,  and  hand 
grenades.  For  each  of  these  duty  areas,  the  available  testing  time  equaled 
the  required  time  to  test  the  full  domain.  The  panels  of  job  experts  also 
made  decisions  concerning  the  testing  time  for  each  duty  area.  First,  ten 
minutes  was  thought  to  be  the  minimum  time  in  which  any  duty  area  could 
be  meaningfully  tested.  Second,  given  the  large  number  of  duty  areas  to  be 
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TABLE  2 


IMPORTANCE  RANKING  AND  TESTING  TIMES  FOR 
DUTY  AREAS  OF  THE  BASIC  INFANTRY  MOS 


Importance 

Training 

Behavioral 

Total 

Testing 

Duty  area 

ranking 

steps 

elements 

time 

time 

Land  navigation 

1 

16 

53 

66 

30 

Tactical  measures 

2 

11 

59 

368 

30 

M16A2  rifle 

4 

15 

56 

240 

30 

SAW 

4 

5 

10 

14 

15 

NBC  defense 

4 

28 

85 

145 

20 

First  aid 

6 

21 

107 

113 

20 

Communications- radio 

7 

10 

31 

90 

20 

Security  and  intelligence 

8 

9 

36 

25 

15 

Grenade  launcher 

9 

12 

41 

69 

15 

Mines 

10 

10 

52 

156 

15 

Night  vision  device 

11 

5 

17 

28 

10 

Communications- telephone 

12 

7 

21 

33 

10 

LAW 

13 

6 

20 

12 

10 

Hand  grenades 

14 

3 

10 

10 

10 

covered,  thirty  minutes  was  established  as  the  maximum  amount  of  time 
to  be  devoted  to  any  one  duty  area.  Finally,  the  duty  areas  were  grouped 
and  time  limits  were  assigned  to  each  block  as  shown  in  table  2.  In  essence, 
the  testing  times  become  relative  weights  of  the  duty  areas  and  also  have 
implications  for  the  scaling  of  the  hands-on  test  scores. 

The  final  sampling  rule  eliminates  some  training  steps  from  being  con¬ 
sidered  in  the  sampling  process.  Essentially,  this  eliminates  many  of  the 
training  steps  that  require  live  firing  of  weapons.  However,  training  steps 
involving  weapon  systems  that  contain  components  other  than  just  firing 
the  weapon  are  not  eliminated,  just  the  specific  component  to  fire  is  deleted. 
For  example,  engaging  the  target  with  the  M203  grenade  launcher  (see  fig¬ 
ure  1)  requires  more  than  firing  the  launcher.  It  is  still  possible  to  test 
clearing  the  M16A2  rifle  and  launcher,  loading  the  launcher,  and  taking 
immediate  action  and  adjusting  sights  based  upon  simulated  feedback  of 
the  firing. 

The  second  eliminating  condition  of  repetitive  or  trivial  steps  is  included 
to  ensure  efficient  use  of  the  testing  time.  Efficiency  implies  maximum  in¬ 
formation  for  the  minimum  amount  of  testing  time.  For  example,  the  total 
time  listed  in  table  2  for  the  tactical  measures  duty  area  is  368  minutes. 
Two  thirds  of  this  time  is  required  for  the  training  step  of  “construct  fight¬ 
ing  hole.”  Certainly,  there  are  required  techniques  and  specifications  for 
the  digging  fox  holes  and  these  concepts  can  be  tested,  but  actually  digging 
a  hole  is  not  efficient  use  of  testing  time. 

Selecting  Test  Content  and  Constructing  Testable  Units 

Selecting  test  content  involves  more  than  randomly  sampling  among 
equivalent  training  steps  within  duty  areas.  Our  strategy  is  to  maximize 
test  content  coverage  across  the  behavioral  elements,  while  not  excluding 
the  possible  replication  of  behavioral  elements.  In  this  manner,  the  empha¬ 
sis  is  placed  on  measurement  of  the  underlying  behaviors  associated  with 
the  performance  of  job  requirements  and  not  on  specific  tasks  or  training 
steps.  Both  dimensions  of  the  behavioral  element  by  task  matrix  (given  in 
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figure  1)  enter  into  the  sampling  process,  not  just  the  task  dimension  which 
has  been  the  customary  practice  in  the  construction  of  hands-on  tests. 

The  sampling  of  training  steps  is  conducted  within  each  duty  area.  To 
maximize  test  content  coverage  across  the  behavioral  elements,  each  train¬ 
ing  step  not  excluded  by  the  previously  discussed  criteria  is  weighted  with 
respect  to  its  behavioral  elements.  Table  3  presents  the  weights  for  a  hy¬ 
pothetical  sampling  of  training  steps  for  the  M203  grenade  launcher  duty 
area.  For  the  first  round  of  the  sampling  process,  the  sampling  weight  for 
each  training  step  is  merely  the  number  of  behavioral  elements  that  it  con¬ 
tains.  Therefore,  the  training  steps  with  the  greatest  number  of  behavioral 
elements  have  the  highest  likelihood  of  being  sampled.  A  training  step  is 
randomly  sampled  from  the  pool  of  weighted  training  steps.  In  our  example 
with  the  M203  grenade  launcher,  the  first  randomly  selected  training  step 
was  “correct  the  leaf  sight.” 

The  selection  of  a  training  step  establishes  the  general  content  area  to 
be  tested,  but  the  training  step  does  not  directly  translate  into  what  will  be 
scored  as  pass  or  fail  on  the  hands-on  test.  At  this  point,  a  thorough  task 
analysis  of  the  training  step  is  conducted  to  refine  the  training  step  into 
a  “testable  unit.”  The  task  analysis  identifies  the  basic  steps  necessary 
to  complete  the  training  step,  including  the  conditions  and  standards  of 
performance,  sources  of  likely  failure,  and  indications  of  exceptionally  good 
performance.  The  task  analysis  is  reviewed  by  job  experts.  Representative 
infantry  job  incumbents  also  review  the  analysis  from  the  perspective  of 
how  they  perform  the  steps.  In  addition,  the  criteria  used  earlier  to  screen 
the  training  steps  are  applied  at  the  step  level  of  the  task  analysis.  That  is, 
steps  of  the  task  analysis  are  deleted  if  they  are  hazardous  or  too  expensive 
to  test  or,  in  the  opinion  of  job  experts,  provide  little  or  no  information. 
Therefore,  the  testable  unit  may  be  somewhat  smaller  than  a  complete 
training  step  and  may  possibly  exclude  part  or  all  of  some  behavioral  ele¬ 
ments.  The  result  of  the  task  analysis  is  a  test  item  for  the  hands-on  test, 
a  testable  unit. 

The  amount  of  testing  time  required  to  complete  the  testable  unit  is 


TABLE  3 


A  HYPOTHETICAL  EXAMPLE  OF  SAMPLING  OF  TEST  CONTENT 
FOR  THE  M203  GRENADE  LAUNCHER  DUTY  AREA 

Panel  A:  Training  Steps  and  Sampling  Weights 


Training  step 

Behavioral 

elements 

Total 

time 

Sampling  weights  by  round 

1 

2 

3 

4 

Visual  inspection 

3 

2 

3 

3 

1 

1 

Operational  inspection 

3 

4 

3 

3 

1 

1 

Maintain  launcher 

5 

32 

5 

5 

0 

0 

Prepare  launcher  to  fire 

3 

3 

3 

3 

1 

1 

Fire  launcher 

2 

3 

2 

2 

2 

2 

Correct  leaf  sight 

2 

3 

2 

0 

0 

0 

Correct  quadrant  sight 

2 

3 

2 

1 

1 

1 

Confirm  zero 

4 

3 

4 

3 

3 

3 

Engage  targets 

7 

3 

7 

7 

5 

5  i 

Emplace  stakes 

2 

3 

2 

2 

2 

o  1 

Fire  -  limited  visibility 

5 

3 

5 

5 

3 

3 

Prepare  Range  Card 

_ 

2 

7 

2 

2 

2 

2  ! 
*  1 

! 

i 

Panel  B:  Randomly  Selected  Test  Content  and  Testing  Times 

Testable  Testing 

Round  unit  time 


1  Correct  leaf  sight  2 

2  Maintain  launcher  8 

3  Emplace  stakes  3 

4  Prepare  range  card  4 


estimated  based  upon  the  task  analysis.  Approximately  2  minutes  are  nec¬ 
essary  to  correct  the  leaf  sight  of  the  M203  grenade  launcher  in  a  testing 
situation.  This  time  is  subtracted  from  the  allocated  testing  time  for  the 
grenade  launcher  noted  in  table  2,  resulting  in  13  minutes  being  available 
to  test  other  content. 

Sampling  weights  for  the  remaining  training  steps  are  recomputed.  The 
two  behavioral  elements  of  the  “correct  leaf  sight”  training  step  are  ex¬ 
cluded  from  entering  into  the  sampling  weights  of  the  other  training  steps. 
The  sampling  weights  for  the  second  round  of  sampling  are  noted  in  table 
3.  Since  the  behavioral  elements  of  the  first  selected  training  step  were 
not  common  to  many  other  training  steps,  the  sampling  weights  are  not 
dramatically  changed. 

The  random  sampling  of  weighted  training  steps  continues  until  the 
testable  units  exhaust  the  allotted  testing  time  for  the  duty  area.  For  the 
M203  grenade  launcher,  the  following  training  steps  were  randomly  selected 
and  refined  into  testable  units  until  the  fifteen  minutes  for  the  duty  area 
were  depleted:  maintain  launcher,  emplace  stakes,  and  prepare  range  card. 
If  complete  coverage  of  the  behavioral  elements  is  achieved  before  the  test¬ 
ing  time  is  consumed;  i.e.,  sampling  weights  for  all  training  steps  equal  zero, 
then  the  sampling  process  begins  anew  with  all  behavioral  elements  con¬ 
tributing  to  the  sampling  weights  for  those  training  steps  yet  to  be  tested. 

Alternate  forms  can  readily  be  produced  by  simply  repeating  the  sam¬ 
pling  technique.  The  resulting  forms  are  parallel,  being  only  randomly 
different.  The  testing  conditions  can  be  changed  if  the  same  training  step 
is  selected  for  both  forms  or  the  testable  unit  within  a  training  step  can 
even  be  redefined. 

SUMMARY 

It  has  been  our  purpose  to  present  an  objective  procedure  for  construct¬ 
ing  a  hands-on  job  performance  test  -  a  procedure  with  a  somewhat  dif¬ 
ferent  focus  than  previously  used  methodologies.  That  is,  we  would  like  to 


be  able  to  interpret  the  resulting  test  scores  in  an  absolute  sense.  It  is  our 
belief  that  a  domain-referenced  approach  is  necessary  for  the  linkage  of  job 
performance  and  aptitude  in  determining  standards  on  the  ASVAB.  This 
absolute  interpretation  of  test  scores,  which  we  called  competency-based 
measurement,  is  based  upon  two  critical  requirements  of  the  test  construc¬ 
tion  process:  detailed  specification  of  the  job  requirements  domain  and 
appropriate  sampling  of  test  content  from  that  domain.  Having  satisfied 
these  two  criteria,  an  individual’s  level  of  competency  is  estimated  by  the 
percentage  of  training  steps  correctly  performed  on  the  hands-on  test. 

Existing  research  by  the  services  provides  an  excellent  foundation  for 
the  specification  of  the  job  requirements  domain.  However,  such  materials 
must  be  supplemented  with  additional  analyses  to  identify  the  behaviors 
associated  with  performing  the  job.  In  this  manner,  the  behavioral  dimen¬ 
sion  serves  as  a  means  to  generalize  across  training  steps,  tasks,  duty  areas, 
and  ideally  across  MOSs.  The  end  product  is  a  behavioral  element  by  task 
matrix  that  completely  defines  the  job  requirements  domain.  This  matrix 
serves  as  the  basis  for  selecting  test  content. 

Any  selection  technique  of  test  content  must  involve  job  experts.  Their 
knowledge  and  expertise  are  critical  throughout  the  entire  process:  from 
defining  the  sampling  unit  and  establishing  the  sampling  rules,  to  refining 
the  training  steps  into  testable  units.  Such  participation  is  necessary  to 
ensure  that  the  resulting  decisions  are  salient,  reasonable,  and  generally 
acceptable.  However,  job  experts  do  not  actually  make  specific  selection 
decisions,  but  rather  they  provide  explanation,  description,  and  definition 
of  different  aspects  of  the  job  requirements.  Having  established  the  bound¬ 
aries  and  limits  among  decision  alternatives,  random  sampling  is  the  most 
appropriate  means  of  making  the  actual  selection. 

Sampling  of  equivalent  training  steps  involves  both  dimensions  of  the 
behavioral  element  by  task  matrix.  Traditional  construction  procedures 
have  tended  to  sample  only  the  task  dimension.  However,  our  strategy 
has  been  to  maximize  coverage  across  the  behavioral  element  dimension  by 
weighting  the  training  steps  in  proportion  to  their  number  of  unique  behav- 


iors.  This  process  does  not  exclude  the  possible  replication  of  measurement 
by  behavioral  elements  but  does  seek  to  cover  as  broad  a  range  of  behaviors 
as  is  feasible. 

It  is  our  belief  that  the  competency-base  test  construction  approach 
described  in  this  paper  is  responsive  to  the  ultimate  purpose  of  the  JPM 
Project:  to  establish  defensible  qualification  standards  on  the  ASVAB  based 
upon  hands-on  measures  of  job  performance.  In  addition,  we  feel  that  the 
consideration  of  the  behavioral  characteristics  of  jobs  will  greatly  facilitate 
the  generalization  of  results  to  MOSs  that  will  not  be  tested. 


APPENDIX  A 


DUTY  AREAS  OF  THE  INFANTRY  OCCUPATIONAL 

FIELD 


TABLE  A-l 


DUTY  AREAS  OF  THE  INFANTRY  OCCUPATIONAL  FIELD 

Basic  Infantryman  (0300) 

1.  Tactical  measures,  11  tasks 

2.  Security  and  intelligence,  4  tasks 

3.  Night  vision  device,  2  tasks 

4.  M16A2  service  rifle,  2  tasks 

5.  M203  grenade  launcher,  5  tasks 

6.  Squad  automatic  weapon  (SAW),  2  tasks 

7.  Light  antitank  weapon  (LAW),  3  tasks 

8.  Hand  grenades,  3  tasks 

9.  Mines,  5  tasks 

10.  Communications,  10  tasks 

11.  Land  navigation,  3  tasks 

12.  First  aid,  2  tasks 

13.  Nuclear,  biological,  and  chemical,  5  tasks 

Infantry  Rifleman  (0311) 

1.  Tactical  measures,  5  tasks 

2.  Squad  automatic  weapon  (SAW),  5  tasks 


TABLE  A-l  (Continued) 


Machine  Gunner  (0331) 

1.  M60  machine  gun,  10  tasks 

2.  M2  machine  gun,  13  tasks 

3.  MK19  machine  gun,  12  tasks 

Mortar  Man  (0341) 

1.  60mm  mortar,  18  tasks 

2.  81mm  mortar,  14  tasks 

Assaultman  (0351) 

1.  Dragon,  4  tasks 

2.  SMAW,  5  tasks 

Infantry  Unit  Leader  (0369) 

1.  Tactical  measures,  3  tasks 

2.  Operations,  5  teaks 

3.  81mm  mortar,  3  tasks 

4.  Mines,  1  task 
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